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Abstract 

A total of 312 classroom teachers responded to a questionnaire about the Reading Framework, a 
document used in developing the 1992 National Assessment of Educational Progress in Reading 
(NAIiP). Considering teachers' reactions is important because the Framework signals a major change 
in the way NAEP assesses reading. The results from the forced-choice items lend considerable support 
for the rationale articulated in the Framework, whereas the comments yield a more complex view of the 
teachers' reactions to it. The questionnaire used in the Undy is appended. 
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WHAT DO CLASSROOM TEACHERS THINK ABOUT THE 
1992 NAEP IN READING? 



The 1990s will probably go down in educational history an a period of great change in assessment 
practices. It will be important for future historians of this period to know something about what 
classroom teachers thought about the bold initiatives undertaken in an effort to advance educational 
assessment. This report should be of use to this end because k provides information about teachers' 
reactions to the conceptual framework developed and used \n designing the 1992 National Assessment 
of Educational Progress in Reading (NAEP). 

Background 

During February 1992, fourth-, eighth-, and twelfth-grade students in randomly selected schools across 
the United States participated in the NAEP in Reading. Mandated by Congress, NAEP is a national 
test given every two years to gather information about students' performance in a number of subject 
areas (e.g., reading, writing, science, and mathematics). The National Assessment Governing Board, 
which administrates NAEP, decides on the subject areas to assess and is responsible for identifying 
appropriate achievement goals for each grade level. The board also sets guidelines for the analysis of 
data and the reporting of results. 

In 1989, the National Assessment Governing Board contracted with the Council of Chief State School 
Officers to develop a rationale and give recommendations for the 1992 NAEP in Reading, In the past, 
NAEP considered the nation's students as a single body and reported its data on the basis of grade level, 
gender, ethnicity, and type of community (rural or urban). In response to recent requests from both 
state and national political leaders, NAEP data are being reported on a state-by-state basis for the states 
that volunteered to participate. In 1992, the trial state assessment applied only to fourth-grade 
performance in reading. 

State-by-state reporting is controversial. Because of this, the Council of Chief State School Officers took 
particular care in planning for the NAEP in Reading. It created the Reading Consensus Project and 
charged it with developing a framework that would represent a consensus among the various 
constituencies in the field of reading. Researchers, practitioners, and state and local school 
administrators were consulted as the framework developed. To this end, drafts of the framework were 
sent out for comment to chief state ischool officers, state assessment directors, school administrators, 
teachers, professors of reading, and assessment experts. The final version of this document, entitled the 
Framework for the 1992 National Assessment of Educational Progress (National Assessment Governing 
Board, 1992) was used to develop specifications for the assessment exercises. 

Because of the importance of developing a sound reading assessment, the National Academy of 
Education commissioned the authors of this report to conduct a study on the content and curricular 
validity of the 1992 NAEP in Reading (Bruce, Osborn, & Commeyras, 1991). One of our charges was 
to determine the degree to which the Framework represented a consensus about reading among reading 
educators. As part of our study, we developed a questionnaire based on the content of the Framework 
and sent it to a diverse group of educators and administrators. The data gathe xd by this questionnaire 
are reported elsewhere (Commeyras, Osborn, Si Bruce, 1992). A subsequent survey was undertaken 
to gain more specific information about classroom teachers' reactions to the Framework. This report 
focuses on the results of the second survey. 
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Survey Respondents 

in August 1991, wc mailed 1,(X)0 questionnaires to a random selection of teachers who were members 
of the International Reading Association (Appendixes A, B, and C contain a copy of ihe questionnaire, 
the cover letter, and an overview describing NAEP and the plans for the 1992 reading a^sessn^ent). We 
received 312 responses. Most of the respondents (84%) were elementary school teachers, the balance 
were middle school, high school, special education, or Chapter 1 teachers. When asked to identify their 
area of specialization, 76% of the respondents identified reading, whereas 15% selected writing. 
y\pproximately half of the respondents said they were "somewhat familiar" with previous NAEPs in 
Reading; 41% said they were "not at all familiar" vnth them. Fifty-three percent of the respondents 
reported teaching for more than 15 years. The profile that seems to fit the majority of these 
respondents is that of an experienced elementary school teacher who has a special interest in teaching 
reading aiid little or no knowledge about NAEP. 

Teachers' Reactions to the Reading Framework 

The questionnaire contained a series of items on the major elements detailed in the Framework. The 
teachers were asked to make forced-choice responses to each item and encouraged to write comments. 
The information garnered from this survey is particularly rich because so many (82%) of the 
respondents wrote comments. The framework implies major changes in the way NAEP assesses 
reading. The teachers' reactions to these changes provide an opportunity to consider what practitioners 
think about some of the current trends in reading assessment. The remainder of this report will look 
at teachers* responses to six of these changes: (a) the use of authentic texts, (b) the use of three 
different kinds of reading situations, (c) the assessment of cognitive aspects of reading, (d) the use of 
open-ended items, (e) the use of special studies, and (f) the practice of state-by-siate reporting of 
results. 

Authentic Texts 

The Framevvotk proposed that the passages in the 1992 NAEP in Reading be full-length authentic texts 
of the kind that students are likely to encounter in everyday reading (e.g., short stories, newspaper 
articles, bus schedules, textbook chapters, pages from telephone directories). It further proposed that 
the passages be much longer than those used in previous NAEP assessments, and not paragraphs written 
solely to as .,ess specific skills. 

The majority of respondents agreed with the idea of using only authentic passages (65%), and they 
presented a variety of reasons for doing so- The reason given most often was that authentic passages 
would test "rear reading: 

/ believe that students should be tested (and instructed) in materials such as those they will be 
encountering in real life situations. 

Other respondents commented on the validity of the assessment: 

By using authentic passages, the validity of the assessment is increased. 

Still others focused on the relationship between assessment and instruction in their comments. Some 
hked the use of authentic passages because it would send the right message to teachers about 
instruction: 

77ie NAEF must come closer to actual classroom reading/writing experiences as reflected in our 
increasing use of a whole language approach to literacy. 



ERLC 



i 



Commeyras, Osborn, & Bruce 



What Do Classroom Teachers Think - 4 



Another set of comments focused on testing for skills with authentic Lexis. Some respondents thought 
reading skills could be tested with authentic passages: 

Specific reading skills can be checked using authentic texts. TJiese are what they will be using 
everyday. 

Others thought testing specific skills was beside the point: 

We've done so mucii assessment of specific skills, yet it hasn^t translated into an understanding 
of how a student derives meaning. Let us pursue authentic assessment. 

Motivation was yet another reason some teachers supported the use of authentic passages. They thought 
students would be more likely to find the passages interesting and thus do better on the assessment: 

/ am so glad that these tests will finally have passages that are meaningful to students - everyday 
reading and real-life experiences are one way to encourage students to put foilh an effort when 
responding. 

About a dozen comments supported using authentic passages, but offered some cautions and conccr.is: 

Authentic passages represent the real world, but I think fiilNength texts would be overwhelming. 
Wlxy not use excerpts of texts? 

I worry about prior knowledge. Kids who know the subject are at an advantage. 

Another group of respondents (33%) favored an assessment that combined authentic passages with 
passages written to test specific skills. Many of these teachers thought doing so would yield a broader 
and more varied assessment: 

A true test takes in consideration competency in a wide variety of situations. 

I believe it is most important that students see the need to be good readers and be exposed to 
relevant passages, Jiierefore the test should be over what they have been exposed to, while at the 
same time some compromise has to be made with incorporating specific skills that will be true 
indicators of how well the child can read. So test what they have been taught but use skills that 
can be measured. 

Three Reading Situations 

The Framework called fur the passages on the 1992 NAEP in Reading to be classified into three types 
of reading situations: (a) reading for literary experience, (b) reading to acquire information, and (c) 
reading to perform a task. It suggested the following plan for assessing these three types of reading 
situations: reading for literary experience by having students respond to questions about short stories, 
poems, or essays; reading to acquire information by having them respond to questions about a magazine 
and newspaper articles, encyclopedia, and textbook selections; reading to perform a task by having them 
respond to questions about bus and train schedules, directions for games, recipes, maps, and so forth. 

More than 80% of the teachers thought it was either "very important" or "absolutely essential" to assess 
reading in thei,e three situations. The strongest support was for reading to perform a task-83% of 
respondents said that this was 'absolutely essential." Following that, 68% of the respondents thought 
it was "absolutely essential" to assess reading to be informed. Interestingly, only 33% of the teachers 
considered reading for literary experience "absolutely essential." 
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Oniy 15% of the teachers commented on the use of the three reading situations. Some comments 
provide explanations for rating reading for literary experience as less than essential: 

Pcifonning tasks is basic to existence for all people, regardless of educational level. Staying 
informed is also important in adult life. Literary experieuce will be primarily for enjoyment, in 
later life. 

As a literature teacher I think reading for literary experience is extremely important, but reading to 
be informed and reading to perfomt a task is essential for all. 

There were also comments from those who rated all three situations as "absolutely essential"; 

/ am especially enthusiastic about this innovation involving assessment in 3 reading situations. 
Again, we must get the word to teachers so this assessment will reflect what is being taught in the 
classroom! 

Other comments included suggestions about the use of the three reading situations: 

/ also feel it would be beneficial to ask students to identify what kind of reading they think they 
need to do. 

Assessment of Cognitive Aspects of Reading 

The Framework proposed that proficient readers use a range of cognitive abilities to construct meaning 
from a text and to elaborate upon and respond critically to it. It identifies four abilities: (a) forming 
an initial understanding, (b) developing an interpretation, (c) personal reflection and response, and (d) 
demonstrating a critical stance. The Framework also proposed that these cognitive aspects of reading 
are not sequential, hierarchical, or representative of subskills. Finally, it proposed assessment of the 
four cognitive aspects in each of the three reading situations. 

It suggested, for example, that the ability to construct meaning be assessed by two types of questions: 
forming-initial-understanding questions, which require students to provide an initial impression or global 
understanding of what they have read, and developing-an^nterpretation questions, which require students 
to go beyond their initial impressions to create a more complete understanding of what they have read. 

The Framework also suggested thai the ability to elaborate on or respond critically to a text be assessed 
by two types of questions: personal reflection and response questions, which require students to connect 
knowledge from the te:rt with their own personal background knowledge, and demonstrating-critical- 
stance questions, which require students to consider a text objectively through critical evaluation, 
comparison and contrast, application to practical tasks, and understanding the impact of such text 
features as irony, humor, and organization. 

More than 85% of the respondents thought that assessing these four cognitive aspects was "very 
important" or "absolutely essential." Persona! reflection and response was selected most often as 
"absolutely essential" (62%). A number of teachers qualified their support for personal reflection and 
response. There were concerns about the role of background knowledge and the relationship among 
the cognitive aspects of reading: 

If the reader has no personal background or prior knowledge, not only will personal reflection and 
response be a zero but the reader will also have absolutely no idea what he/she just read. 
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rm not sur* developing an interpretation and personal response and reflection can be considered 
apart - these work for the reader in tandem. 

Another set of cornmcnts focused on demonstrating a critical stance. A number of teachers questioned 
the appropriateness of this kind of reading: 

Cognitively, this may be a little heavy for fowih graders. 

TJxis must be measured carefully considering the developmental level of the reader (child to adult) 
and depends largely on the reader's use of "a" (forming an initial understanding). 

Not all reading requires this - Sometimes it's just for enjoyment. 

Open-Ended Items 

The Framework called for approximately 40% of ap>sessment time to be spent on open-ended items, 
substantially more time than was given to such items in any previous NAEP assessment. The rationale 
given for increasing the number of open-ended items was that it would evaluate a reader's ability to 
generate organized, thoughtful responses to authentic reading passages. 

Virtually all of the teachers supported the inclusion of open-ended items (97%). Almost as many agreed 
with the rationale given for including open ended items (94%). Most teachers (76%) thought 40% of 
the test-taking time period was about the right amount of time for students to spend on open-ended 
items. The remaining teachers thought it was either too much (8%) or too little (11%). 

Twenty-four percent of the teachers commented on the inclusion of open-ended items. Many of these 
comments focused on the perceived benefits of open-ended items over multiple-choice items: 

Tlxere has been far too much emphasis placed on multiple choice exams and regurgitation of 
'facts." Open-ended items encourage emphasis on the development of higher level thinking skills. 
Ws been my experience that very little in this world is either black or white. 

Important to explore and consider students' responses as they more closely reflect thought processes 
and reasoning than do other types of questioning (i^e., multiple choice). 

Other teachers who favored the inclusion of open-ended items brought up concerns about their use: 

/ appreciate the inclusion of open-ended items. Reliance upon written responses is logical, but 
many students will not produce the level of organized thought/response unless given an opportiuiity 
to read and revise their written responses. 

Tlie only problem is that, ultimately, you are judging reading ability through writing (the written 
response), and that isn't always an accurate way to evaluate. 

I think it is important that this fonn of assessment not be done too early - i.e. be sure it is 
developmentally appropriate for age group, 

I fear the subjectiveness of interpretation. 

These four comments address areas of concern that seem worthy of further consideration. The first 
relates to how the test will be administered, and whether a match exists between test administration and 
classroom practices that encourage students to engage in process writing. The second comment notes 
that open-ended items confound reading and writing abilities. The third suggests that assessing reading 
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through writteni responses should be handled differentially by grade leveL The last comment hints at 
the issue of how to score open-ended items fairly. Thus, the great enthusiasm for open-ended items 
indicated by these teachers was coupled with substantial procedural concerns. 

Special Studies 

The Framework called for the 1992 NAEP in Reading to collect additional information about the 
reading performance of fourth-grade students through special studies, called the oral reading, portfolio, 
and metacognitive studies. The respondents were asked to indicate the extent to which they thought 
each study was needed. 

Only 15% of the teachers thought the oral reading study, which was to assess fluency by timing and 
analyzing students' oral reading, was greatly needed. Approximately 9% of the respondents thought the 
study was "not at all" needed. There were more comments on this study than there were for the other 
two studies. Most of the comments were objections to assessing oral reading: 

/ hope you would NEVER time oral reading. Wty? Wlxat is the purpose? 

Oral reading is seldom done in real life and can be affected by too many factors unrelated to 
actual reading skills. 

Some people are not fluent orally - it's not important to be orally fluent to be a good reader. 
Silent reading skills are cruciaL 

Other teachers wanted more information about the methodology to be employed in the oral reading 
study: 

How will this be measured? How will objectivity be maintained? I believe thai it is ven^ important 
but wonder how, realistically, this can be measurea. 

Is the focus fluency? Perhaps careful error analysis would provide information as valuable as just 
a measure of fluency. 

More than half the teachers (56%) judged the portfolio assessment study, which was to gather and 
analyze examples of actual classroom work in reading, as well as to gather information about what 
students read in class and on their own, as very important. Less than 3% of respondents regarded the 
portfolio assessment study as unnecessary. The teachers comments indicate they differ in their views 
about the benefits of studying portfolio assessment. Some were quite enthusiastic: 

Portfolio assessment is the answer because it "shows" more than any grade or test the strengths and 
weaknesses of each individual student. 

Will allow us to consider reading in a broader sense^ beyond the testing day. 
Other teachers foresaw potential difficulties: 

"Portfolio'' is a bandwagon, and it is being {and will be} misused, I think. Proceed carefullv, 
PLEASE! 

Portfolio Assessment would be time consuming and not necessarily more accurate than group test. 
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The metacognitive study was to examine readers' awareness of their own comprehension. Their use of 
effective reading strategics was assessed, analyzed, and reported as descriptive data. This study received 
strong support from 52% of the teachers. Only 3% thought it was superfluous. Comments on this study 
were split between those who thought it would yield useful information and those who were skeptical: 

As one involved in metacogniiive instruction I ant panicularly heanened by this emphasis. 

I'm a good typist. But to tty to explain what I do when I type would be venf difficult. It's become 
too much a part of me. I think a good reader has much the same problem. 

State-by-State Reporting 

The last item on the questionnaire asked about the decision to report NAEP data at the state level. The 
teachers reactions were mixed on this issue. The majority of teachers indicated they "strongly favored" 
(37%) or "somewhat favored" (35%) state-by-state reporting. Fewer said they were "somewhat opposed" 
(15%) or "strongly opposed" (10%) to it. Only a handful (3%) had no opinion. Teachers "strongly in 
favor" of state-by-state reporting made comments like the following: 

As a teacher, my cuniculum is mandated by the state. If our children compare poorly to other 
states, it will give the "powers that be" son. ' "food for thought." 

Only because n might biing about some changes in instniction - even more important it might help 
the legislature to appropriate more money for education. 

Those teachers who were only "somewhat in favor" of state-by-state reporting articulated concerns about 
how the results will be reported and used in the name of educational reform: 

Demographic statistics could be included to help inteqyret. For example^ I teach in a school in 
a lower socioeconomic area where education is not valued to the cMent it is elsewhere. I hate to 
see my students* scores compared to those of an area where student backgrounds encourage higher 
achievement. 

It's nice to know where we rank in relation to other states: however I become leery of 
administrators, news coverage, and politicians who misinteqyret and sensationalize the results. It's 
too bad that they do not have global understanding of what they read. 

Those who were "somewhat opposed" or "strongly opposed" expressed similar concerns: 

Such reporting may lead to misinterpretations on the part of the public. States at the bottom may 
feel defeated and students graduating from high schools in such states may find it more difficult 
to be accepted by universities. Business people contemplating establishing offices and factories in 
such states may decide to go elsewhere. 

I dislike claims based on test results that one area is doing a better job of educating students than 
another. I'd rather know how what you find in your portfolio study relates to test scores. Or - how 
does giving students some choice in what they read relate to test scores? 

The comments indicate thai these teachers are inclined to qualify their support for state-by-state 
reporting. They seem to have a whole host of fears regarding the misuse of test data and seem deeply 
concerned about the potential harm that might be caused by this kind of reporting. 
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Conclusions 



Any conclusions drawn from the results of this survey questionnaire should be made in light of two very 
important considerations. The fust is that the survey concerns the Framework, the document that 
articulates the rationale for the 1992 NAEP in Reading, The respondents did not see any test passages 
or items. It is entirely possible that their support for the elements of the Framework would not 
necessarily hold for the actual assessment. The fit between the Framework and the assessment is a 
separate issue that we have addressed elsev.here (Bruce, Osborn, & Commeyras, 1991). 

The second consideration concerns the characteristics of the 312 teachers who responded lo the survey. 
Primarily, these were elementary school teachers who belong to the IRA, These teachers were apt to 
be more aware of current trends and issues in reading because they receive as part of their IRA 
membership Hie Reading Teacher, a journal reporting current theory, research, and practice on literacy. 
The reactions of the teachers who filled out our questionnaire may be representative of elementary 
school teachers who belong to IRA but not all elementary teachers. 

Our conclusions are made with these two considerations in mind. Overall, there is considerable support 
from many teachers for the rationale articulated in the framework for the 1992 NAEP in Reading, 
More teachers favor exclusive use of authentic passages than those v*..o want passages written to test 
specific skills. Assessing reading to perform a task and reading to be informed are perceived as more 
essential than assessing reading for literary experience. There is strong support for the four cognitive 
aspects of reading that were used to develop questions that assess constructing, extending, and examining 
meaning. There is virtual unanimity for including open-ended test items but differences of opinion 
about the proportion of opeuTended to multiple-choice items. In regards to the special studies, there 
is far more enthusiasm for investigating portfolio assessment and metacognition than there is for 
assessing fourth graders' reading fluency. Lastly, there are more teachers who favor state-by-state 
reporting than oppose it. 

The comments made by the respondents to this questionnaire demonstrate that what looks fairly 
straightforward in percentages becomes far more complex in light of the comments accompanying the 
forced-choice options. The teachers used comments to qualify responses by voicing concerns, posing 
questions, and offering contextual information. We find chese data valuable because they provide a 
window through which to view teachers' thoughts on what matters when evaluating reading performance. 
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APPENDIX A 



REACTIONS OF CLASSROOM TEACHERS TO NAEP READING FRAMEWORK 



Survey Based on the Content of the Framework for the 1992 NAEP (312 Respondents) 
A. Characteristics of Proficient Readers 

1. The Framework identifies characteristics of proficient, or good, readers. To what extent do you 
agree or disagree with each of the following statements about these characteristics. 

Strongly Strongly 

a gree Agr ee Disagree disagree 

a. Good readers possess the knowledge, 
behavior, and attitudes that allow for 

continual learning through reading . . 79.49% 19.23% 0.32% 0.64% 
(No response - .32%) 

b. Good readers read with enough 
fluency so that they can focus on the 
meaning of what they read, rather 

than devoting a lot of attention to 85.49% 13.78% 0.32% 0.64% 

figuring out the words 

c. Good readers use what they already 

know to understand the text they are 84.29% 14.74% 0 0.64% 

reading 

(No response = .32%) 

d. Good readers extend, elaborate, and 66.99% 30.45% 1.60% 0.96% 
critically judg€ the meaning of what 

they read 

74.35% 23.70% 0.65% 0.32% 

e. Good readers plan, manage, and 
check the progress of their reading . 

(No response = .32%) 45.83% 44.23% 8.65% 0.96% 

f. Good readers use a variety of 
effective strategies to aid their 

understanding 82.05% 16.67% 0.32% 0.64% 

(No response = .32%) 

g. Good readers can read different types 
of texts and can read for different 

purposes 82.69% 16.03% 0.32% 0.64% 

(No response = .32%) 
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B. Views of Reading 



2. The Framework makes the following statements about reading. Indicate the extent to which you 
find each definition or statement acceptable. 



a. Reading is a complex process that 
involves an interaction among the 
reader, the text, and the context, or 
situation 

b. The term "reading literacy" connotes 
more than basic or functional literacy. 
Specifically, it connotes knowing when 
to read, how to read, and how to 
reflect on what is being read 

c. Proficient reading is essential for 
successful functioning in schools, 

homes, and workplaces 

(No Response = .32%) 

d. Proficient reading contributes to a 

sense of personal satisfaction 

(No Response = .64%) 



Very 
acceptable 



81.73% 



69.87% 



76.60% 



75.32% 



Acceptable 



17.31% 



28.85% 



22.44% 



23.72% 



Very 

Unacceptable unacceptable 



0.64% 



0.96% 



0.64% 



0.32% 



0.32% 



0.32% 



C. Reading Situations 

3. In the NAEP, students' reading ability will be assessed in three situations: reading for literary 
experience^ reading to be informed, and reading to perform a task. In your opinion, how 
important is it to assess students' reading ability in each situation. 

Absolutely Very Moderately Somewhat Not at all 
essential important important important important 

a. Reading for literary experience (short 

stories, poems, essays) 33.01% 50.96% 14.42% 1.60% 0% 

b. Reading to be informed (magazine 
and newspaper articles, encyclopedias, 

textbook chapters) 68.27% 29.49% 1.60% 0.32% 0.32% 

c. Reading to perform a task (bus and 
train schedules, directions for games, 

recipes, maps, etc.) 82.69% 15.06% 1.92% 0^ 0.32% 
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4a. The following table shows the proportion of items in the NAEP allocated at each grade level to 
each type of reading situation. This distribution of items is intended to reflect the changing 
demands made of students as they progress through school. 



Grade 
level 


Reading for literary 
experience 


Reading to be informed 


Reading to perform a 
task 


4 


55% 


45% 


0% 


8 


40% 


40% 


20% 


12 


35% 


45% 


20% 


Do you agree with this NAEP allocation of items? 

Yes 39.74% 

No 58.65% 

No Response 1.60% 

4b. If you disagree, use the table below to show how you would reallocate the proportion of items. 


Grade 
level 


Reading for literary 
experience 


Reading to be informed 


Reading to perform a 
task 


4 


46.32 (7.37) 


38.60 (6.87) 


15.05 (8.67) 


8 


38.92 (6.75) 


38.92 (5.24) 


22.13 (6.95) 


12 


35.07 (7.82) 


40.81 (6.67) 


24.17 (8.14) 
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D. Cognitive Aspects of Reading: 



5. The Framework identifies four cognitive aspects of reading to be assessed within each of the 
three rr^ding situations. Each of these aspects is described below. Indicate how important you 
believe t '.h to be. 

Absolutely Very Moderately Somewhat Not at all 
essential important important important important 

Forming an initial understanding 
requires the reader to provide an 
initial impression or global 

understanding of what was read . . . 50.64% 41.67% 7.37% 0 0 

(No Response = .32%) 



Developing an interpretation 
requires the reader to develop 
a more complete understanding 
of what was read by linking 
Information across parts of a 
text as well as by focusing on 
specific information in the text . 
(No Response = 0.32%) 



52.26% 



42.95% 



4.17% 



Personal reflection and response 
requires the reader to connect 
knowledge from the text with his 
or her own personal background 
knowledge 

(No Response = .32%) 



61.86% 



27.88% 



9.29% 



0.64% 



Demonstrating a critical stance 
requires the reader to consider 
the text objectively and involves 
a range of tasks including critical 
evaluation, comparing and contrasting, 
application to practical tasks, and 
understanding the impact of such text 
features as irony, humor, and 

organization 48.40% 38.46% 9.94% 2.24% 0.32% 

(No Response = .64%) 

6a. In your opinion, are there other cognitive processes of reading that need to be represented in the 
Framework? 

Yes 9.29% 

No 80.45% 



No Response 



10.26% 



E. Open-ended Items 



7a. Approximately 40% of the assessment time will be spent on open-ended Items. This is 

substantially more time than has been given to such items in any previous NAEP assessment. 
The following rationale is given for the increased use of open-ended items is that they provide a 
means for examining whether readers can generate organized, carefully thought-out responses to 
reading. Also, open-ended items more cL;sely resemble the real world tasks that students must 
perform outside of school. 

Do you support the inclusion of open-ended items in the NAEP Assessment? 



Yes 
No 



97.12% 
1.28% 



No Response 



1.60% 



7c. Do you agree with the rationale given above for including more open-ended items? 



Yes 
No 



94.23% 
1.60% 



No F^esponse 



4.17% 



7d, Do you think the proportion of open-ended items 



(40%) in the NAEP is: 



Too much 

The right amount . . . . 
Too little 



8.33% 

76.60% 

10.58% 



No Response 



4.49% 
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F. Passage Selection 



8a. The passages used in the NAEP will be authentic, full-length texts that students are likely to 
encounter in everyday reading (i,e,, short stories, newspaper articles, bus schedules, textbook 
chapters, pages from telephone directories). The passages will not be paragraphs written solely 



to assess specific reading skills. 
In your opinion, should the assessment use . . . 

Only authentic passages, 65.38% 

Only passages written to test 

specific skills, or 0,64% 

A combination of authentic 
passages and passages written 

to test specific skills? 32.69% 

No Response 1.28% 



G. Teaching to the Test 

9. Although the Framework claims that assessment should not drive instruction, it also states that 
the NAEP assessment must be an appropriate guide to instruction, should teachers use :t to 
"teach to the test." 

a. Do you believe that NAEP should attempt to develop an assessment that can serve as a 
useful guide to instruction? 

Yes 71.47% 

No 25.32% 

No Response 3.21% 
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H. SpecM Studies 



10. Several types of Information about the reading performance of students will be collected from 
special studies with small subsamples of students. These studies are listed below. To wnat 
extent do you believe that each of these studies is needed? 



a. Oral Reading . Fluency will be 
assessed by timing and analyzing 
students' oral reading 



Not at all 

1 2 



To a very 
great extent 
4 5 



8.01% 



23.72% 



41.03% 



19.55% 



7.69% 



b. Portfolio Assessment . Portfolio 
activities will be used to gather 
and analyze examples of actual 
classroom work in reading as 
well as to gather information 
about what the students read in 

class and on their own 

(No Response = 0.32%) 



1.60% 



4.49% 



9.29% 



39.10% 



45.19% 



c. T he Metacognitive Study. 

Readers* awareness of their own 
comprehension and their use of 
effective reading strategies will 
be assessed, analyzed and 
reported as descriptive data 
(No Response = 0.96%) 



2.56% 4.49% 



35.58% 



42.31% 
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L Goals of the 1992 NAEP in Reading 

11. The committees that developed the Framework were given the following set of guidelines. 
Indicate how well you believe the assessment will meet each one of the guidelines. 

To a very 

Not at all great 
well extent 

a. To focus on outcomes (is 1 2 3 4 5 
performance oriented), rather 

than representing an instructional 

or theoretical approach 0.64% 2.24% 23.08% 43.91% 25.00% 

(No Response = 5.13%) 

b. To address changing literacy 

needs for employability, personal 0.64% 3.85% 14.74% 42.63% 33.65% 

development^ and citizenship . 
(No Response = 4.49%) 

c. To expand the scope of 
assessment strategies by including 
open-ended questions and special 
studies on oral reading, portfolio 

assessment, and reading 0.64% 2.24% 6.41% 33.33% 53.53% 



strategies 

(No Response 3.85%) 

d. To reflect contemporary research 

on reading and literacy 1.28% 2.56% 12.50% 42.31% 36.86% 

(No Response = 4.49%) 

e. To provide information for policy 
makers and educators that will 
assist in the improvement of 

educational performance .... 0.64% 2.24% 15.38% 42.63% 34.29% 

(No Response = 4.49%) 



J. State»bV'State Reporting 

12. The 1992 NAEP for reading will provide state-by-state as well as national reports of student 
performance. How do you feel about state-by-state reporting of student performance? 

Strongly in favor 36.54% 

Somewhat in favor . . . 35.26% 

Somewhat opposed . . . 15.06% 

Strongly opposed .... 9.94% 

No Response 3.20% 
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K. Personal Information 

13a. Are you currently employed as a teacher at the elementary, secondary, or college level? 

Yes 94,87% 

No 4.49% 

No Response 0.64% 

b. Indicate the grade level(s) you teach. (CIRCLE ALL THAT APPLY.) 

K-2 45.19% 

3-5 38.46% 

6-9 1538% 

10-12 3.85% 

Special Education 4.17% 

Chapter 1 4.17% 

Undergraduate 2.56% 

Graduate 4.81% 

Other (SPECIFY) . . . 3.85% 



14a. Do you currently hold an administrative position at the school, district, or state level? 



Yes 6.73% 

No 90.38% 

No Response 2.88% 

b. What administrative position do you hold? 

Superintendent of Public Instruction 0% 

Superintendent/Assistant Superintendent 0% 

Principal/Assistant Principal 2.56% 

Reading Coordinator/Supervisor/Consultant . . . 4.17% 

Other (SPECIFY) 1,60%. 



15a. Do you currently hold a college or university position? 

Yes 7.05% 

No 90.06% 

b. What position do you hold? 

Professor 1.60% 

Administrator 0% 

Research Associate ... 0% 

Other (SPECIFY) . . , 5.45% 



16. Indicate the area that best represents your field of specialization. 

Reading 75.%% 

Writing 15.38% 

Assessment 5.45% 

Other (SPECIFY) . . . 16.35% 

17. How many years have you been employed in the field of education? 

0-5 years 13.46% 

6-10 years 13.78% 

1M5 years 19.23% 

16-25 years 36.54% 

Over 25 years 16.67% 

18. How familiar are you with previous National Assessments of Educational Progress in Reading? 

Very familiar 6.41% 

Somewhat familiar . . . 51.92% 
Not at all familiar .... 41.35% 
Thank you very much for your cooperation. 
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COVER LETTER 



May 1, 1991 
Dear Educator: 

We are writing to ask for your help as we prepare a review of the 1992 National Assessment of 
Educational Progress (NAEP) in Reading. The reading NAEP for 1992 will be based on the ideas 
and recommendations set forth in a document called the Framework, The National Academy of 
Education has asked us to examine this document to determine the extent to which it represents a 
consensus about reading among professionals in the field of education. 

The review we prepare will be one of ten such reviews of various aspects of the 1992 NAEP that 
have been commissioned by the National Academy. These reviews will form the basis of a report 
that will be presented to Congress in October. 

So that the ideas and recommendations of the Framework couid be commented on by a number of 
educatorji, we have prepared a survey questionnaire based on the content of the Framework, We 
hope very much that you will find the survey of interest, and that you will complete it., and return it 
to us. Most questions can be answered by circling a single code number following the question. Wc 
encourage you to comment on any or all of the items that are of particular importance to you. 

To provide some background information, we have enclosed a brief overview thai describes the 
NAEP, the development of the Framework, and some special features of the 1992 reading 
assessment. If you have any questions, please call us at 217-333-6551. 

We would be most appreciative if you could return the survey in the envelope provided by May 31, 
or otherwise, at your earliest convenience. 

Your responses and comments will be invaluable to us as we prepare our review for the National 
Academy. Thank you very much for your time and your effort. 



Best wishes. 



Bertram Bruce 
Professor of Education 
University of Illinois 



Jean Osborn 

Associate Director 

Center for the Study of Reading 



Michelle Commeyras 

Project Associate 

Center for the Study of Reading 



Janet Salm 

Project Assistant 

Center for the Study of Reading 
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OVERVIEW OF NAEP AND PLANS FOR 1992 ASSESSMENT IN READING 

What Is The National Assessment of Educational Progress? 

The National Assessment of Educational Progress - "the Nation's Report Card" is 
mandated by Congress. Every two years, NAEP assesses the performance of more than 
120,000 fourth-, eighth-, and twelfth-grade students in the nation's schools. The purpose is to 
gather information about students' performance and about changes in their performance 
over time. 

Since 1969, NAEP has conducted seven assessments in reading, six each in science and 
mathematics, five in writing, two each in music and art, and one in computer science. 
NAEP has also conducted special assessments in citizenship, U. S. government, U. S. 
history, literature, social studies, and other areas. 

The National Assessment Governing Board decides on the subject areas to be assessed, 
including those specified by the Congress. It also is responsible for identifying appropriate 
achievement goals for each age level; for developing the objectives, specifications, and 
procedures for each test; for setting the data analyzing and reporting guidelines; and for 
determining procedures for interstate, regional, and national comparisons based on the data. 

The Framework for the 1992 NAEP 

In 1989, the National Assessment Governing Board contracted with the Council of Chief 
State School Officers to develop a rationale and give recommendations for the 1992 
National Assessment of Educational Progress for Reading. Because of the diverse and often 
conflicting opinions about reading and its assessment held by reading educators and others 
in the field of education, the Council created the Reading Consensus Project and charged it 
with developing an assessment framework that would be acceptable to the field as a whole. 

In response to this charge, the Reading Consensus Project appointed committees composed 
of teachers and administrators, members of state departments of education, university 
professors whose specialties incluaed reading and assessment, and representatives from a 
number of educational, business, and professional organizations. These committees met 
between October 1989 and February 1990. 

The n^embers of these committees were dedicated not only to developing a framework that 
would reflect the consensus of the field of reading but also to ensuring that the framework 
would be consistent with sound, contemporary research about reading. To this end, drafts 
of the developing framework were sent out for comment to a large number of chief state 
school officers; state assessment directors; school administrators and teachers; professors of 
reading, education, and psychology; and assessment experts. The committees' final version 
of the Framework was submitted to the National Assessment Governing Board in June 
1990. 
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Changes from Earlier NAEP Assessments 



The Framework proposes some major changes in the 1992 NAEP assessment: 

L Authentic Texts. The assessment will use reading passages drawn from books and 
articles like those students read in school and on their own, rather than passages written 
solely for testing purposes, such as for assessing particular reading skills. These passages 
will be much longer than those used in previous NAEP assessments. The eighth-grade 
students will, for example, read an entire short story, a n'^.wspaper article, and a complete 
set of instructions. 

2. Three Reading Situations. The passages students will read are classified into three types 
of reading situations: (1) reading for literary experience, (2) reading to acquire information, 
and (3) reading to perform a task. Reading for literary experience will be assessed by 
having students respond to questions about a short story or a poem. Reading to acquire 
information will as assessed by having them respond to questions about a newspaper article 
or a textbook selection. Reading to perform a task will be assessed by having them respond 
to questions about an instruction manual or a train schedule. 

3. Assessment of the Cognitive Aspects of Reading. The Framework recognizes that 
proficient readers use a range of cognitive abilities to construct meaning from a text and to 
elaborate upon and respond critically to it. The Framework also recognizes that these 
cognitive aspects of reading are not sequential or hierarchical and do not represent a set of 
subskills. The Framework proposes that these cognitive aspects be assessed within jach of 
the three reading situations. 

The ability to construct meaning, for example, will be assessed by Uvo types of questions: 

Forming initial understanding questions, which require students to provide an initial 
impression or global understanding of what they have read. 

Developing an interpretation questions, which require students to go beyond their initial 
impressions to create a more complete understanding of what they have read. 

The ability to elaborate on or respond critically to a text will also be assessed by two types of questions: 

Personal reflection and response questions, which require students to connect knowledge 

from the text with tiheir own personal background knowledge. 

Demonstrating critical stance questions, which require students to consider a text 

objectively. 

4. Multiple-Choice and Open-Ended Questions. Approximately 60% of assessment time will be spent 
multiple-choice questions, 40% on open-ended questions. Some of the open-ended questions will be 
designed for one- or two-sentence answers, others for more extended written responses. Primary-trait 
scoring will be used for the extended responses, and scoring rubrics will be created for each question. 
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5. Tliree Special Studies 

Several types of information about the reading performance of students will be collected from special 
studies with subsamples of students. 

Oral Reading . Tape recorded interviews will be used to examine the oral reading fluency of fourth- 
grade students. 

Portfolio Assessment . Taped interviews will also be used to gather information about classroom 
reading instruction. For these portfolio type activities, the students will talk about both their 
independent and classroom reading assignments. In addition, they will be asked to bring samples of 
their wriiten work to the interview. 

Metacognitive Study . The metacognitive strategies (the strategies readers use to monitor their 
reading comprehension) that fourth-, eighth-, and twelfth-grade students employ as they read will be 
investigated. 

6. State-by-State Reporting, In the pas', NAEP considered the nation's students as a single body, and 
reported its data on the basis of grade level, gender, ethnicity and type of community (rural or urban). 
In response to requests from both state and national educational leaders, NAEP data now will also be 
reported by state. The 1990 NAEP mathematics assessment will provide state-by-state information, and 
the 1992 fourth-grade reading assessment will do the same. It is anticipated that in the future, all NAEP 
data will be reported as both a national and as state-by-state assessments. 



